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Abstract 

In this paper we develop a finite blocklength version of the Output Statistics of Random Binning (OSRB) 
framework. The framework is shown to be optimal in the point-to-point case. New second order regions for broadcast 
channel and wiretap channel with strong secrecy criterion are derived. 

I. Introduction 

Output Statistics of Random Binning (OSRB) is a new framework for proving achievability results 0]. It works 
by converting channel coding problems into source coding problems, and uses the induced pmf of the source coding 
side to design encoders for the channel coding side. The goal is to make the total variation distance between the 
joint pmf of the source coding side and channel coding side close to zero so that all the performance analysis 
can be dealt with at the source coding side where Slepian-Wolf (S-W) theorem can be invoked. Thus the OSRB 
technique is not based on the usual covering and packing lemmas. 

Originally studied by Strassen [2|, there has been a recent surge of works on finite blocklength information theory 
following the work of Polyanskiy et al (3) (see for instance In this paper we develop a finite blocklength 

version of the OSRB framework. We show that this method is optimal in the point-to-point channel and can directly 
give us the channel dispersion E[Var(?,(X; We also use the technique to derive the second order region 

for broadcast channel (that recovers Marton's inner bound in the asymptotic case) and for wiretap channel with 
strong secrecy criterion (that improves the result of []8]). Scenarios such as broadcast wiretap channel can be also 
dealt with using this technique but have been left out for a more complete version of this draft. 

OSRB is based on two theorems: the S-W theorem and another theorem that may be considered as its dual. To 
develop a finite blocklength version of the OSRB, we first find a one-shot version of these two main theorems. By 
one-shot we mean that only a single use of the resource is allowed. To get finite blocklength results, we then apply 

This work was supported by Iran-NSF under grant No. 88114.46. 

1 Direct proofs for this formula have also been obtained by Wang et al. using a different technique 1 6 1 . 
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this result to a product of n use of the network. The resulting dispersion at the output can be either due to the 
dispersion in the input code or to the inherent dispersion of the channel. To avoid the input dispersion, we use a 
uniform distribution over a fixed type in the source coding side of the problem. In this sense this differs from the 
original asymptotic OSRB where we use a completely i.i.d. distribution in the source coding side of the problem. 

This paper is organized as follows: some definitions and notations are given in Section [II] One-shot version of 
the two main theorems of the OSRB are given in Section [III] We then apply the technique to a couple of problems 
in Section [IV] To illustrate the use of the technique we begin by recovering the known result on dispersion for 
the point-to-point channel in Subsection II V-AI In Subsections IIV-BI and II V-Cl we apply the technique to broadcast 
channel and wiretap channel. 

II. Definitions 

Definition 1: Given a pmf px.Y, me conditional information of x given y is defined by 

h p (x\y) := log 5——. 

Px\Y{x\y) 

Also, the information density i p (x]y) is defined by 

/ \ , p(x,y) 
i p {x;y) := log———. 

p{x)p(y) 

Definition 2: Let X be a multi-dimensional normal variable with zero mean and covariance matrix V. The 
complementary multivariate Gaussian cumulative distribution region associated with V is defined by 

C _1 (V,e) :={x: P(X < x) > 1 - e}. 

Notation: In this paper, we use X\> to denote (X v : v € V) and p v A to denote the uniform distribution over the 
set A. The total variation between two pmf's p and q on the same alphabet X , is defined by \\p{x) — (/(x)^ := 
I \p( x ) ~ l( x )\- When a pmf itself is random, we use capital letter, e.g. Px- See JT] Remark 1] for more details 
about random pmfs and their manipulations. 

III. One-shot Output Statistics of Random Binning 

Let (Xy, Z) be a set of discrete sources distributed according to a joint pmf px v .z on a finite set (Iluev ^u) x -Z. 
A distributed random binning consists of a set of random mappings B v : X v —> [1 : M„], v G V, in which B v maps 
each sequence of X v uniformly and independently to the set [1 : M^]. We use B v as a shorthand for rv B V (X V ), and 
By or £?y(Xy) as a shorthand for rv (B v (X v )) ve x>. A distributed random binning induces the following random 
pmE on the set X% x Z n x ILevI 1 : M ^]' 

P(xv,z, b v ) = p{x Vl z) l{B v (x v ) = b v }. (1) 

2 The pmf is random due to the random binning assignment in the protocol. 
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The asymptotic OSRB introduced in Q] relies on the S-W theorem as well as Theorem 1 of H) that implies 
independence of random bin indices under certain conditions. To set up a non-asymptotic framework, we generalize 
the S-W theorem and Theorem 1 of 0] to the case of a single channel use. Let us begin with the latter: 
Theorem 1: Given px v .z, for any pmf tz and any positive real 7, the random pmf of eq. ([TJ satisfies 

E ||P(& V , z) - p u (b v )p(z)\\ 1 < p XvZ (S,(p\\ty) + 2^"\ 

where the expectation is over the randomness of binning and the set <S 7 (p||f) C Ay x Z n is defined as follows: 

&y(p\\t) :={(x v ,z) :V$^S CV,h p (x s ,z)-h t {z)-J2 l °g M v > l)- 

ves 

Remark 1: This theorem implies [1 1 Theorem 1] by evaluating it for the product px™,z™ — TYi=iPx v it Zi- Set 

|V| — 1 — 7 

tz n = Pz™ and 7 = nS for a sufficiently small value of S > that we discuss later. Then the term 2 2 
converges to zero. The first term converges to zero as well if S 1 (p\\t) includes almost all of the typical set T p . For 
any jointly typical (xy, z"), the terms h p (xg, z n ) and h p (z n ) are close to nH p (Xs, Z) and nH p (Z), respectively. 
Thus as long as H p (Xs\Z) > — J2ves logM„ we can choose 7 = nS for a S > such that the inequalities defining 
S 1 (p\\t) holds. 

Remark 2: The rv Z in the statement of the above theorem is of use in problems with secrecy constraints. 

Proof: See Appendix lAl ■ 
One shot S-W coding: Here we want to bound the error probability of decoding a single copy of the source X\> 
when the decoder has access to the side information Z as well as the bin indices By. An optimal decoder uses 
ML decoding. However we use an stochastic variation of MAP for the decoding with a more tracktable analysis. 
The decoder draws xy from the conditional pmf Px v \z,b v {xv\Vi ^v)> where P is the induced probability by the 
random binning. More specifically 

D / » I in p(xy\z)l{B v (x v ) = b v ) 

^X v \Z,B v {XV\Z,0 V ) = — J-^-- 

We refer this decoder as a stochastic likelihood coder (SLC). See ifTTl for a motivation of SLC and the justification 

for using a stochastic decoder. For some technical reasons, we can more generally use a mismatch SLC corresponding 

to an arbitrary pmf tx v ,z instead of p in the above expression^] that is, 

rr /- 1 ■ n t(x v \z)l(B v (x v ) = 6y) 

1 x v \z,B v {x v \z,b v ) = — — — — — . 

' v J2x v t{x v \z)l{B v (x v ) = by) 

Roughly speaking, the reason for introducing a mismatch SLC is that we will need to work with input codewords 

of the same type to reduce the total dispersion, rather than with codewords generated from an i.i.d. distribution. 

However we need independence to be able to use the Berry-Essen CLT at a later stage. A mismatch SLC allows 

us to simultaneously employ an independent and a non-independent distribution. 

3 The pmf tx v ,z should not be confused with the one used in Thm[T]where it is only defined on Z. 
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Theorem 2: Given px v ,z and any pmf tx v .z, the expected value of the probability of correct decoding of a 
mismatch SLC associated with t is bounded from below by 

EP\C] > E„ — 1 = , , y lyc ^ , (2) 

P l + T, U s^v M s^ ht(Xs]XsZ) 

where Ms = Hues Moreover, this bound can be weakened to give the following bound on the error probability 
of mismatch SLC, 

EP[£] <Px v z(S 7 (i M ) c ) + (2 |v| - 1)2^, (3) 
where 7 is an arbitrary positive number and 

S 7 (ix v ,z) := {(x V , z) : V0 ± S C V, ^ log M, - /i t (x 5 |z) > 7}. (4) 

Remark 3: Using this theorem one can derive finite blocklength analogs of the S-W theorem for i.i.d. or non- 
i.i.d. sources. Since we choose the codewords from a fixed type, we use this theorem in its non-i.i.d. form. I.i.d. 
forms of the S-W theorem have been previously obtained by [7j. 

Proof: We only prove the inequality for the special case of |V| = 1. For the complete proof, see Appendix 
IB] The probability of correct decoding can be written as, 

P[C] = E P( x > z)HB(x) = b)T x \ ZtB {x\z, b). 

x,b,z 



We have, 



t(x\z) 

R 

t{x\z) 



EP[q = E £ Pi x, ,)!(*(,) = b) ^ (5) 



MEgp(M)W = l) £it(e| ^ ) = 1) 
„„v^ ^ , * = 1) 



B t(s|*)l(fl(2) = l) 



(6) 
(7) 



E {B(*),S^}Ei*(i»F)l(5(i5) = l) 



t(x\z)\(B(x) = 1) 

*^}E. t(«l*)i(s( 
/ \ t(x\z)l(B(x) = 1) 

- £ E ^ «) ^ 1Z )W,)'ii);M- 1( i-^i 2 )) 



(9) 
(10) 



where (O is due to the symmetry, (|8) follows from the Jensen inequality for the convex function f(x) = - on the 
R + and (fTOb follows from the fact that B(x) and B(x) are independent for any i/i. ■ 

IV. Applications of non-asymptotic OSRB 

To illustrate the use of the tools introdued in the previous section, we recover a finite blocklength result for the 
point to point channel coding, and prove new results for broadcast channel and wiretap channel. Since the structure 
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of the proofs are similar, we have tried to provide a detailed proof for the simplest case, i.e. the point-to-point 
channel and outline other proofs have less details. See [1 1 1 for the full proofs. 

A. Point to point channel coding 

Consider a DMC channel qy\x- We will recover the result of [3 1 that there is an (n, e)-code with rate 



in the first step we obtain a one-shot achievable rate following the OSRB technique. In the second step we use 
Theorem Q] and Theorem [2] for the n uses of the channel, to approximate the achievable rate. 

Step 1: One-shot OSRB: Just like the asymptotic OSRB, the first step is itself divided into three parts. In 
the first part we start from a source coding problem, use random binning and then find an upper bound on the 
error probability. In the second part, we use the joint pmfs of the source coding side of the problem to design a 
concrete encoder-decoder for the channel coding with one exception: the encoder-decoder is assisted with a common 
randomness that does not really exist in the model (to be removed in third part). We will find upper bounds on 
the total variation distance of the joint induced pmfs between all r.v.'s in the two parts. The bounds on the error 
probability of S-W coding and the total variation distance of the joint induced pmfs give a bound on the error 
probability of encoder-decoder of the part two. In the third part, we eliminate the common randomness given to the 
second protocol without disturbing the probability of error. This makes the designed encoder-decoder in the second 
part useful for code construction. 

Part 1: Source coding problem and random binning: We start from a different problem of source coding; we 
will use the pmf induced by this problem to construct our channel code in the next part. Let (X, Y) be distributed 
according to q(x,y) = q(x)q(y\x). We define two random mappings on X as follows: to each x, we assign two 
random bin indices m 6 [1 : M] and / € [1 : F], uniformly and independently. This induces a joint pmf on M, F, X 
which we denote by P s (m, f, x). Suppose that the decoder chooses a tx,Y and uses a mismatched decoder T(x\y, /) 
constructed using tx.Y- Then the induced random pmf is P s (x, y, m, f, x) = q(x, y)P s (m, f\x)T(x\f, y). Invoking 
Theorem |2 with rv Z being a constant, one can derive an upper bound epec on the expectation of error probability 
that only depends on F (and not on M). This upper bound is provided later in equation dTTt for the finite blocklength 
coding. 

Part 2: Designing encoder-decoder assisted with a shared randomness: Returning to the channel coding problem 
we assume that there is a shared randomness F available at both the encoder and decoder, which is independent 
of the message and uniformly distributed over [1 : F]. This shared randomness does not exist in the original setup 
and we will eliminate it later. The encoder uses the conditional pmf P s (x\m, /) of the source coding problem. The 
decoder uses the mismatched decoder T(x\y, /) to find x and thereby an estimate of the message rh. The induced 
random pmf is P c (x,y,m, f,x) = p u (m, f)P s (x\m, f)p(y\x)T(x\f,y). We have 




(12) 



for any arbitrary input pmf qx where V — E [Var gy , x (i{X; Y)\X)\ . Our framework is divided into two steps: 



\\P s (x,y,m,f,x) - P c (x,y,mJ,x)\\ l = 1 1 P s (m, /) - p u (m, /) 



(13) 
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Given M and F, Theorem Q] gives an upper bound 6a px on the expectation of the total variation distance between 
P s and P c . Observe that using P c instead of P s changes the probability of error by at most £& px . Thus the expected 
error probability EgP[£] of the channel coding is bounded above by eoec + ^Apx- 

Part 3: Eliminating shared randomness: Using the law of iterated expectation, we have EgP[£] = Ee i?P[£|F] < 
epec + £Apx- Thus there exists a fixed binning and an instance /* of F, such that the encoder p s (x\m, /*) and the 
mismatched decoder T(x\y, /*) results in a pair of encoder-decoder with error probability of at most epec + £Apx- 

Step 2: Non-asymptotic analysis: We would apply the one shot OSRB bound to n i.i.d. repetitions of the DMC 
qy\x- I n IDi we started from an i.i.d. input for the source coding part. Although using an i.i.d. distribution makes 
evaluation of Theorem Q] and Theorem [2] simple, but this does not yield an optimal strategy. This is due to the 
fact that an i.i.d. input causes a dispersion in addition to the inherent dispersion of the channel. To avoid input 
dispersion, we choose channel input sequences with the same type. 

Let M = 2 nR and F = 2 nR . For a given qx and n, we can find a n-type <f>x such that the infinity norm 
\\&)P ~ 1x\\co < ^. To prove ( TTSI i. assume that the px™ is a uniform distribution over the set 71(n) of sequences 

(n) 

with the type $ x . The known bounds on the size of typical sets imply that there exists L such that for sufficiently 
large n, log|71(n)| > nH.^fX) — Llogn. Setting 7 = logn, Z a constant and |V| = 1 in Theorem [T] gives the 
following bound on the right hand side of equation dZTT i and thus on 6a px : 

e Apx <p x „(^) + -^, (14) 
V n 

where we have used the theorem with X n being the Xy in the statement of the theorem. Further 

5 7 := {x n : h pxn (x n ) - log n > n(R + R)}. (15) 
Note that for each x n € 71(n) the relation h p n (x n ) = log 171 1») | holds. Hence if we set 

n{R + R) =nH, (n) (X) - (L + 2)\ogn, (16) 

then the first term of (fl4l vanishes and we have eadx < ~^=- 

v n 

Next we should find R such that the error probability epec < e — The decoder has access to Y n and a single 
bin index F of X n . Setting 7 = -logn, |V| = 1, Z = Y n as well as F as a bin index of X\> = X n in the 
statement of Theorem [2] we get that for any tx^Y™, we have 

£Dec < P x^q Y -\xAS(t) c ) + 4=. ( 17 ) 

V n 

where S(t) :— {(x n ,y n ) : nR— h t (x n \y n ) > - logn}. Observe that Yi,--- , Y n are conditionally independent 
given any X n = x n because the channel is memoryless. So if we can write ht(x n \y n ) as a sum of independent 
rv's, we would be able to use Berry-Essen CLT to find R. Using tx n Y n — Px n qy n \x n does not give rise to 
such a factorization. To overcome this situation we use ix™,Y™ = qx n qY n \x n — I\i=i l( x i)Q(yi\ x i) f° r mismatch 
decoding. We then have h t (x n \y n ) = >~27=i hqfailVi)- Given X n = x n , {h q (xi\Yi)}™ =1 are functions of independent 



March 5, 2013 



DRAFT 



7 



rv's, and hence mutually independent; thus we can now apply the Berry-Essen CLT to bound the first term of (ITTt . 
Using the Berry-Essen CLT for each x n £ 71(n), we have 

q Y ~\x~= x ™ {S{t) c ) = PG (G >nR~\ logn) + 0(-£=) 
where G is a normal r.v. with 

n 

EG = ^2E qY .^.h q (xi\Yi) = = x]E qY ^h q (x\Y) 

i—1 x 

= J2n^ n Hx)E qYlx h q (x\Y)=nE^ ) E qYlx [h q (X\Y)\X] 
VarG = XVar^^CxilYO = nE^)Var gy|X K(X|Y)|X] 

i=l 

= nE $ ( ?) Var g , |x [* g (X;F)|X]. 
The sketch of the rest of the proof is as follows (see [10] for details): analyzing the bound (fTTT i, we get that 



R = nE, {n) E qY , x [h q (X\Y)\X] + /nE. ( „, Var gy|x r)|X]Q- x (e) + O(logn), 



(18) 

is sufficient to achieve e Dec < e — -7=. Now ||$^ - gx||oo < ^ implies that = H q (X) + O(^), 

E^,E gy|x [VX|y)|X] = ff fl (X|y) + O(i) and E^j Va W ^(X;Y)|X] - U g + 0(±) yields Finally 
combining these relations with ( fT6] l and ( fTSl imply (1121 1, 



B. Broadcast channel 

Consider the problem of transmission of two private messages over a broadcast channel qY x Y 2 \x- Let TZ*(n, e) 
be the set of all rate pairs (Ri, R^) of all (n, e)-codes where e is the probability of erroneous decoding at either of 
the decoders. We prove a one-shot version of Marton with two auxilaries. A similar theorem is proved for Marton 
with common message and involving auxiliary rv Uo in [10|. 

Theorem 3: Given any pmf quiU 3 x> let Tl;n(qu 1 u 2 x , n, e) be the set of all pairs (Ri, R2) for which there exists 
reals R\ , R2 > such that 

logn x 



Ri + Rj < H q {JJj) - 0(-^-), 3 = 1. 2, 

Rx + R 2 + Ri + R 2 < H q (UiU 2 ) ~ Of- 5 -) 

n 



+ Q- 1 (V BC . 9 , e ) + 0(^), 

n 



Ri 


e 




R2 







(19) 
(20) 



where the entropies are computed according to the pmf qu 1 u 2 XY 1 Y 2 = qu 1 u 2 xqY 1 Y 2 \x and 

V BC , q =E qUiU2 Cov qYiY2lUiU2 [(^(17 X ; Yi), *,(ET a ; K 2 )) T |CT 1 Z7 2 ] . 

Then U qUiU2X TZ in (q UlU2 x,n,e) C K*(n,e). 

Sketch of the proof: The proof follows in similar steps as in the proof of channel coding. 
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Part 1: Source coding side of the problem and random binning: Let (Ui, U2, X, Y) be distributed according to 
q{u\,U2,x,yt,y-i) = q(u 1: U2,x)q(yi,y2\x). Consider the following random binning 

• For j = 1, 2, to each Uj assign independently two random bins rtij G [1 : Mj] and fj £ [1 : Fj]. 
Suppose that the decoder at the receiver j = 1,2 uses a mismatched decoder Tj(v,j\yj, fj) to generate tij and 
thereby rhj . The induced random pmf is 

P s (mi :2 ,TO1:2, A:2, 2/1:2, "1:2) = ^("1:2 , 2/1:2 )P S Ol ) fl |«1 ) 

^("12, /2|w2)? 1 i(ui|/i, 2/i)T 2 (zt 2 |/2, ya)< (21) 

We find an upper bound on EP(«i 7^ «ior u 2 ^ U2) which in turn bounds the probability of error. Using the first 
bound of Theorem 12 and the union bound, we have (see [10] for proof): 
Lemma 1: For any mismatched decoders Tj,j = 1,2, 

EP[£] <Pu 1 .. 2 Y 1 .A^(tut 2 ) c ) +4 x 2-t, (22) 

where 7 is an arbitrary positive number and 

S 7 (t 1 ,t 2 ) := {(ui:2, 2/1:2) : logFj - htjiujiyj) > 7, j = 1,2}. (23) 

Par/ 2; Designing encoder-decoder assisted with a shared randomness: Assume that there is a shared 
randomness (Fi,^) available at the both encoders and the decoder, which is independent of the message and 
uniformly distributed over [1 : Fi] x [1 : F 2 ]. The encoder uses the conditional pmf P s (mi:2, x\mi-.2, fv.2) 
of the source coding problem. The decoder j uses the mismatched decoder Tj(v,j\yj, fj) to find Uj and as 
a result an estimate rhj of the message. Then the induced random pmf is P c (ui:2, x, yi-.2, mxa-, /i:2> *&i:2) = 
P U {mi-.2, fi-.2)Ps(ui:2,x, 2/1:2= "i:2|"ii:2, /i: 2 )- We have 

\\Ps-Pc\U - ||P s (m 1:2 ,/ 1:2 )-p c/ (m 1:2 ,/ 1:2 )|| 1 . 

The probability of error is no more than \\P S — P c \\i and thus no more than the right hand side of the above 
equation. Given M and F, Theorem Q] gives an upper bound 6Apx on the expectation of the right hand side. Observe 
that the expected error probability EgP[£] of the channel coding is bounded from above by epec + £Apx- 

Finally we can eliminate the shared randomness Fi :2 as in the proof of the channel coding. 

Step 2: Non-asymptotic analysis of one-shot OSRB: We would apply the one shot OSRB bound to n repetitions 
of the BC q Yl:2 \x- 

Let Mj = 2 nRj and Fj = 2 nRj . Following the proof of channel coding, for a given qx and n, we find an n-type 
$ij^ 2 such that ||$[^ 2 — ?Ui 2 ||oo < — • To prove ( flZb . assume that pu? 2 is a uniform distribution over the set 71 <>.) 

Cl:2 

of sequences with the type <j>[^ ' . Observe that has a uniform distribution over the set 7^(n) of sequences with 
the type $[7 . ^ s a res uh, h Pu „ (itf) = log |71(») |. As in the proof of channel coding, we can utilize Theorem [TJ 

1:2 Uj 

1 

to show that if the inequalities in (fT9b are satisfied, then 6a px < 0{—=). 
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Next we should find Ri,R.2 such that epec < e— 0(^=). We can utilize Lemma [T] to show that ( f20l > is sufficient 
for eoec < e — ^(^n)' r '^ le res * °^ me P ro °f i s similar to that of channel coding but uses a generalized version of 
Berry-Essen CLT for the independent and multidimensional r.v.'s j9]- ■ 

C. Wiretap channel with strong secrecy 

Consider a wiretap channel with probability transition qyz\x> i n which the receiver and the wiretapper have access 
to channel outputs Y and Z, respectively. For a given (n, R) code we use total variation distance \\pmz™ — v\iVz n || 1 
to measure the security of the code, where pMz n i s me induced pmf by the code. A rate R is said to be (e r , e sec )- 
achievable if there exists an (n,R) code such that P[£] < e r and \\pMZ n — V U MVz n \ x < £sec- 

Theorem 4: Given qy,z\Xi f° r an y input distribution qu,x an d any 9 E [0, 1], the following rate is (n, e r , e se c)- 
achievable: 

R(n, e r , e 5ec ) = J, (17; Y) - I q (U; Z) - v^Q- 1 ^) - V^Q^(0e sec ) - O(^), (24) 

where = 1 - 6, Vy = E qcrx Var qYlux [I q (U;Y)\U] and V z is defined similarly. 

Sketch of proof: For simplicity, we prove the theorem for the special case U = X. We will find R(n, e r , e 5ec ) 
such that EP[£] < 9e r and E ||pa/z™ — PmP^" i < ^ £ sec- Then by Markov inequality, we can find a code with 
the desired conditions. 

One-shot OSRB: We use the same code construction of subsection IIV-AI Here we need to compute the security 
index of the code. To do this, we bound the security constraint, i.e. e SiSec = E||P s (m,/,^) — p u (m, f)q(z)\\ v 
using Theorem Q] in the source coding part of the problem. Then using triangular inequality, the security 
constraint of channel coding asserted with shared randomness e Ci5ec = E P e (m, /, z) — p u (m, f)q(z)\\ 1 is 
bounded above by e S)S ec + £ Apx- To eliminate F, we can show that there exists an instance / such that 
E B|F=/ \\Pc(m,z\f) - p u (m)P c (z)\\ 1 < e s , S ec + 3e Apx - 

Non-asymptotic analysis of one-shot OSRB: Again we follow the analysis of pt-to-pt channel. It was observed 
that if ( fToT ) is satisfied, then 6a px < 1/^/n. Similar error analysis shows that EP[£] < 9e r provided that 

nR = nH q {X\Y) + \AWQ _1 (fer) + O(logn). (25) 

Next we find a constraint on R and R such that security index e s . 5ec < 9tsec — 3/y/n, which shows that 
E||^mz« -PmPZ"-^ < 0e sec . Substituting 7 = logn in Theorem Q] gives: e SiSec < p x »(S°(p\\t)) + l/y/n, 
where 

S 7 (p\\t) := {(x n ,z n ) : h Pxn (x n ,z n )-h t (z n )-logn > n(R + R)}. 

Again as in the proof of error probability for channel coding, to apply Berry-Essen CLT we need to write 
h Pxn (x n , z n ) — h t (z n ) as a sum of independent r.v.'s. Let t(z n ) = J\i=i Qz(zi), which makes h t (z n ) as sum 
of independent r.v.'s. Next observe that for any x n <E 7~<s> xn , Px n Qz n \X n i xn : z ") = qx n ,z n (x n , z n )2~°( logn \ 
Using this fact, we have h Pxnqznix „ (x n , z n ) = h q (x n , z n ) + O(logn); thus 

n 

S y ( P \\t) :={(x n ,z n ):J2 h i( x i\ z i)- ( lo En)>n(R + R)}. 

i=l 



March 5, 2013 



DRAFT 



10 



Applying Berry-Essen CLT in the same way as in channel coding proof implies 



n(R + R) = nH q (X\Z) - y/r^zQ-^de^) + O(logn). (26) 
Combining (|25]l and (J2S1> yields d24)). ■ 
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Appendix A 
Proof of TheoremQ] 

We modify the proof of Theorem 4 of H] to obtain the one-shot version of OSRB. Observe that 

EP(z,b v ) = 1 p(z)=p u (bv)p(z). 

llvEV N ' v 

We can decompose this sum into two parts as follows: 

P(z 7 b v ) = ^2 p{xv,z) Y[ l{B v (x v ) = b v }, 

x v :(x v ,z)<£S~,(p\\t) vEV 

P(z,b v ) = ^ P(xv,z) J} l{B v (x v ) = b v }. 

x v :(x v ,z)£S 1 (p\\t) »€V 



We then have 



Thus 



P(z,bv) = P(z,b v ) + P(z,b v ). 
EP(z, by) = EP(z, b v ) + EP(z, b v ). 



Using triangle inequality we have 



E 



P(z,b v )-p z (z)l[p u (b 



•i 

vev 



= E\\P(z,b v )-EP(z,b v )\\ 1 

i 
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< 



P(z,b v ) -EP(z,b v ) 



• E 



P(z,bv) -EP(z,b v ) 



(27) 



Therefore we need to show that both of the terms on the right hand side converge to zero as n converges to infinity. 
For the second term we have 



E 



z,b v 



(28) 



= E P{z,x v ,b v ) 

z,x\? 7 by.(z,xx>)(£$'Y {p\\t) 

= E ^ P(z,x v ) 

z,x v :(z,x v )^S^ (p\\t) 

= E Y P( z ,xv) 

z,x\>:{z,x\!)^S-y (p\\t) 

^ PXvZ ((s 7 ( P \\t)r), 

where the factor i in the first equation comes from the definition of total variation distance. In the first step we 
use the inequality E\X — EX\ < 2E|X| and in the last step, we use the relation P(z,xv) — Px v z(z,xv) since 
P(z,xv) is not a random pmf. 

Next consider the first term of the r.h.s. of equation (|27V Using Cauchy-Schwarz inequality we have 

E \\P(z, b v ) - EP(z, fcy)^ = \ Y, E \ p ( z > M - Ep (z, Ml < \Y1 \l™{P{z,bv))- (29) 

z,b v z,b v 

Using the formula var(J^Lj Xi) — Yli<i j<r cov (Xi, -Xj) and the fact that P(z, fey) is sum of several terms 

P(z,b v )= P {x Vl z)Y[l{B v (x v ) = b v }, (30) 

x v :(x v ,z)£S 1 (p\\t) l)£V 

we can write 

var(P(z,6 v )) = 



E 



(x v ,j)es T (p||t), (x v ,2)es T ( P ||t 



E 



cov(p(x v , z)l{B v (x v ) = b v },p(x v ,z)l{B v (x v ) = b v }j 
p(x v , z)p(x v , z)cov( l{B v (x v ) = & v }, HB v (x v ) = b v }j (31) 



xy.xy: 

(x v ,z)es 7 ( P ||t), (x v ,2)es T ( P ||t) 



To evaluate the covariance term, we need to find the places where Xy and xy match. We first partition the set 

{(xv,xv) : (xy,z) £ S y (p\\t), {xv,z) £ S 1 (p\\t)} into the sets Af(s,z) defined below: 

Af(S, z) := |(a;v,5v) -Xs = %t 7^ x t ,\/t £ S c , 

(x v ,z) £ S~f(p\\t), (x v ,z) £ 5 T (p||i)|. 

We note that for each pair (iv,iv) inside the set A/"(0, z), the random variables l{B^\xv) = fey} an d 
\{By l \xv) — fey} are independent, thus the covariance between them is zero . Next for S ^ 0, we bound above 
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the covariance for the pair (xy , xy) € Af(s, z) as follows, 

COV (l{4 n) (xy) = b V }, l{B|; n) (xy) = 6y}) < E \{B^ {x V ) = B$\x V ) = b v } 

= E l{4 n) (x s ) = b s ,Bg>M = Bg>(x s *) = b s *} 
1 



(32) 



M 5 M| c ' 

where in the last step, we use the fact that the random variables l{Bg l \xs) = bs}, l{Bg e (%S c ) = &s c } and 
l{Bgc (xs°) = bgc} are mutually independent. 

Since the union is taken over xy and xy in the set 5 7 (p||t), we need to get access to the set S that appears in 
the definition of S 1 (p\\t). For this reason for any subset S we define 

S 7 (pp)(s) := J (x s ,z) : h p (x s ,z) - h t (z) - ^ log M„ > 7 , L . (33) 

I vGS J 

Then from the definition of S 1 (p\\t) we have <S T (p||t) = Hscyi^v : £s € S-y(p\\t)( s )}- Substituting ( l32t in (l3~TT l 
gives 

var(P(z,6y)) < ^ ^ p(xy , z)p(xy, z) - 

0/SCV (a;v,xv)eA/'(S,«) 5 S " 

E m^4|: E p 2 (^^) 

^ p(xs°\xs,z)p(xs°\xs,z) (34) 



< 



X 



(35) 



= E E p 2 (^*) 

< y m 1,2 e p(is > z)t(«) 1 ^-2-'' 

fl^SCV 15 5 x s :(x S :z)eS 1 (p\\t)(S) 

* E m^E^)^2- (36) 

< ^2 v ->(z)t(z) (37) 

where ( |34l follows from relaxing the restrictions xy € <S 7 (p||t) and xy € S 7 (p\\t) while keeping the constraint 
xs = x s € <5> 7 (p||i)(s). Eq. (f33T > follows from p(xs,z) < i(z)-j^-2 _7 which is due to the definition of the set 
S 7 (p\\t)(s). 

Substituting (O in (f29]l yield 



\ \l^{p{ Zl b v )) < E ^ 2 ^ _1 Vp(W) 

z 

V-7 

< 2 2 _1 , (38) 

where (|38l follows from Cauchy-Schwarz inequality. 



March 5, 2013 



DRAFT 



13 



Appendix B 
Proof of Theorem[2] 

First we prove the inequality (0 following the proof for the special case |V| = 1. We have 



Consider, 



p [C] = X! p(xv,z)l(B v (x v ) = b v )Q Xv \z,B v (xv\z,bv)- 

X\? ,6y t Z 



ip[c]=e J2 p{xv,z)i(B(x v ) = bv) = t S£mr- \ I \ 

.^JX&v Sx v t(x v \z)l(B(x v ) = by) 

t(x v \z) 



1 V E p(x V ,z)l(B(x v ) = l v )=— 

x v ,z 2^ix v 



Sv t{x V \z)l(B(x V ) = ly) 

t(x v \z)l(B(x v ) = l v ) 



(39) 
(40) 



Emm / \ ^li-l-Dlxyi — ly; 

^^(xv)^(,v), S v^v}K^V^) Esvt( - vk)l(jB( - v) = ly) (41) 



. M V-m , s ^y|z)l(i?(xy) = ly) 

^{B(x v ),S v ^x v }J2x v t ( X v\z)l(B(x V ) = l V ) 

= y p(xv , z) *£vj£) (44 ) 



" Ep l + E^cyM^2^(^l^^)' (45) 
where (l40l is due to the symmetry and (1421 follows from the Jensen inequality for the convex function f(x) = ^ 
on the To obtain (|43T >. we partition the tuples in the set Ay according to its difference with the tuple X\>. 
Define Ms '■= {xv ■ x~s<= — xs<=Mv e5: x v ^ x v }. Then Ay = UscvM's and for each x\> € Ms, we h ave 

E{B(x v ),x v ^x v }HB(x V ) = ly) = M^l^^s.) = l S c), 

where we have used the fact that [B(x v ) : v G S] and B(x\>) are mutually independent. This implies ( |43T l. 
WeaAxn/ng f/ie bound ©: 

l ^ lpv^eS,^)) 



l((Xy,Z) GS^tx^z)) 



> Ep " ^2^±v^ (46) 
1 + Efl^SCV 2 7 

l((X v ,Z)e5 7 fa,z)) 
p 1 + (2V - 1)2-7 



- 1 + (2V _ i)2-7 PXyZ ( 5 t(^v,z)) , 
where (PfoT i follows from the definition of 5 7 (ix v ,z)- Finally we have 



(47) 
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P[£] = 1 - P[C] 

< Px v z (S^tx^zY) 1 + (2 vl 1)2 - 7 ) l5-fA v .z)l (48) 

<Px v z{S 1 (t Xv ,z) c )+ (l ' 



1 + (2 V - 1)2-t / 

< p Xv z (5 7 (tx v ,z) c ) + (2 V - 1)2-*, (49) 



where (|48j follows from (PiTT i. 
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